Skip to content

Expose WAL-G runtime settings in BackupConfig#67

Open
3loymag wants to merge 1 commit into
wal-g:mainfrom
3loymag:walg-config-overrides
Open

Expose WAL-G runtime settings in BackupConfig#67
3loymag wants to merge 1 commit into
wal-g:mainfrom
3loymag:walg-config-overrides

Conversation

@3loymag

@3loymag 3loymag commented May 12, 2026

Copy link
Copy Markdown

Description

This PR adds optional WAL-G runtime settings under BackupConfig.spec.walg.

Previously, several WAL-G runtime parameters were only configurable through hardcoded defaults in NewConfigWithDefaults(), for example WALG_COMPRESSION_METHOD, GOMAXPROCS, TOTAL_BG_UPLOADED_LIMIT, WALG_FAILOVER_STORAGES_*, and WALG_PREVENT_WAL_OVERWRITE.

This makes operational tuning harder because optimal values may depend on database size, CPU limits, S3/MinIO performance, network limits, WAL generation rate, and backup/restore requirements.

With this change, users can configure these settings declaratively through a BackupConfig manifest, for example:

spec:
  walg:
    compressionMethod: brotli
    uploadConcurrency: 16
    uploadDiskConcurrency: 8
    networkRateLimitBytesPerSecond: 536870912

All new fields are optional. Existing behavior is preserved when `spec.walg` is not set. Existing top-level fields remain supported for backward compatibility.

Internal integration settings such as `PGHOST` and `PGUSER` are intentionally not exposed.

No related issue.

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [x] Documentation update
- [ ] CI/CD or infrastructure change

## How Has This Been Tested?

Tested locally with:

```sh
go test ./api/v1beta1 ./pkg/walg
make generate
make manifests
  • Unit tests
  • Integration tests
  • Manual tests

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@3loymag 3loymag requested a review from a team as a code owner May 12, 2026 10:10
@Endevir

Endevir commented May 18, 2026

Copy link
Copy Markdown
Member

Hi @3loymag! Thanks for your contribution!
Actually most of added fields are already present as part of BackupConfig spec, especially those which are used for tuning between performance and resource constraints (like upload rate limits and disk concurrency), so adding the same fields under different section is discouraged. Some of params (like failoverStoragesCheck and all related) are omitted intentionally because we do not support defining failover storage currently, instead using multiple BackupConfigs and placing backups in different places is preferred.

Talking about WALG_COMPRESSION_METHOD - despite the fact that wal-g allows changing compression type, we make use of brotli as the most battle-tested in our environments one, and it gives suitable balance between speed and compression rate and should be enough for clusters up to several terabytes. And GOMAXPROCS should also be dynamically configured according to sidecar container limits, so there should be no reason to tune it manually.

Can you define which limits do you run into, so we can help you to resolve them or define params that are actually required?

@Endevir Endevir left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to resolve questions in #67 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants